GBIF Data Processing and Validation

GBIF (Global Biodiversity Information Facility) is the largest data aggregator of biological occurrences in world. was officially established 2001 and has since aggregated 1.8 billion occurrence records from almost 2000 publishers. relies heavily on Darwin Core (DwC) for organising it receives. Data Processing Pipelines Every single record that gets published to goes through a series three processing steps until becomes available GBIF.org. source downloading parsing into verbatim interpreting values Once all are standard form, they go set interpretations. In 2018, underwent significant rewrite order improve speed maintainablility. One main goals this consistency between GBIF's Living Atlases. connection with this, current validator fell out sync pipelines processing. New Validator The service allows anyone GBIF-relevant dataset receive report syntactical correctness validity content contained within dataset. By submitting validator, users can validation interpretation procedures usually associated publishing quickly determine potential issues data, without having publish it. planning rework because does not exactly match Planned Changes new will project. Validations be saved show up user pages similar way downloads derived datasets appear now (no more bookmarking validations!) A downloadable found produced. Suggested Changes/Ideas guiding philosophies interface avoiding information overload. often quite verbose its feedback, highlighting may or fixable particularly important. will: generate map geolocations; give by importance; "What", "Where", "When" flags priority; some possible solutions suggested fixes flagged records. We see hosted portal environment as implement pre-publication interactive visual. Potential Quality Flags team been compiling list quality flags. Not easy implement, so cannot promise get implemented, even if great idea. advantage any flag step validator. Easy flags: country centroid : Country/province centroids known problem. zero coordinate Sometimes publishers leave either latitude longitude field when should have left blank NULL. default uncertainty meters flag: value code used dwc:coordinateUncertaintyInMeters, which might indicate incorrect. This especially case 301, 3036, 999, 9999. no higher taxonomy Often record. cause problems matching backbone taxonomy.. null There discussion encourage fill dwc:coordinateUncertaintyInMeters. every record, ones taken Global Positioning System (GPS) reading, an dwc:coordinateUncertaintyInMeters It also nice escape hatch, such publisher rid false positives remove filling value. Batch-type validations doable pipelines, but probably include: outlier: Outliers generally two types outliers: environmental outliers distance outliers. Currently type outlier. sensitive species: species would where considered vulnerable way. Usually due poaching threat only one area. gridded dataset: Rasterized common GBIF. These location pinned low-resolution grid. already experimental API (Application Programming Interface). Conclusion moving targets. Variable always issue aggregating large amounts data. With architecture, we hope features added easily. Time staffing resources short supply, plan prioritise feedback publishers, them work correcting most important issues. projects like vocabulary server, community participation.